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ABSTRACT 

This manual was designed for the administrator who 
has the responsibility for conducting all or part of an evaluation or 
is responsible for using evaluation results in program planning. The 
manual provides step-by-step guidelines to help: identify the 
purposes and audiences of an evaluation; prepare a basic description 
of the program or activity; refine educational objectives and 
establish a priority for the evaluation of each; describe resources 
and processes to be used in achieving^ objectives; specify alternative 
decisions to be made about a program; st^ate evaluation questions; 
establish guidelines so that the evaluation will be consistent with 
the demands of funding agencies, district policy, local concerns and 
ethical principles; identify available resources; specify data 
sources; determine appropriate measures of processes and outcomes; 
apply guidelines to measurement instruments; establish and apply 
criteria for the selection of an evaluation specialist; prepare a 
basic plan for collecting,, analyzing, and reporting information; make 
judgments concerning types and formats for evaluation reporting, and 
use various types of evaluation findings. The manual can be 
supplemented by the Workshop Leader's Guide (TH 007 178). 
Bibliographical references are appc.nded* (Authcr/JAC) 



* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 
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FOREWORD 

o 

Thfs^manLial is written for you, the busy administrator, who has the 
responsibility for conducting all or part of an evaluation or is responsible 
for using evaluation results in program planning. The manual is designed to^ 
assist you in planning and carrying out evaluation of an activity or program. 
It will also help you to work with evaluators when appropriate and to use 
evaluaticp re*='/lts more effectively. 

Administrators in many school districts do not have, the luxury of ., 
program evaluation performed by a full-time evaluation specialist- Even in 
Targe school districts, evaluation specialists are not always available. 
Nevertheless, administrators need to make informed decisions about 
educational activities and programs: in their schools. The information and 
practice provided in this manual s^.ould help you, tMe administrator t better 
evaluate and improve your programs. 

To be of maximum benefit but. of rganageable size, the i:cope and treatment 
of the content of this manual has been limited. It is hoped that there is 
sufficient material to allow you to reach the goals listed below 
and also to encourage you to apply what you have learned and to seek 
further information about evaluation. 

This manual will provide you w^ith step-by-step guidelines to: 

• identify the purposes and audiences for your evaluation 

' • prepare a basic description of your program or activity''' 

• refine educational oibjectives and establish a priority for 
the evaluation of each 

• describe resources and processes to be used in achieving 
your objectives 

• specify al ternative decisions likely to be made about a 
program 

• state evaluation questions 



An activity is considered here as a single function or event such as a 
two-day teacher workshop. A program is considered as a set of activities 
systematically organized to achieve specific outcomes such as a reading 
program in a. school. 



vi ■ ■ /■ • 

• establish evaluation guidelines to make your evaluation 
consistent with .the demands of funding agencies, district 

^ - policy, local concerns and ethica^' principles 

- . • identify available resources for conducting an evaluation • 

, ' • specify data sources 

• determine appropriate ways to measure selected processes 
- / ^ . and outcomes \ 

• .. apply the guidelines stated. in the manual to measurement 

instruments 

o establish and apply criteria for the selection of an 
evaluation specialist 

' • prepare a basic, evaluation plan for collecting* analyzing 
and reporting information 

^ • make judgments regarding various types and formats for 

evaluation reporting 

• use various types of evaluation findings 

The forms -discussed in this manual are available in a separate packet 
which can be used as you plan and conduct any evaluation with which you. may 
be associated. 



<?-- 
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PURPOSES FOR EVALUATION 



Why Bother to Evaluate Soinethln:g ? There, are many 
.^.answers to this question. After reviewing some 
. frequen^t purposes for evaluation^ it is useful for 
you to decide, in a specific sifuation, why you ;are 
doing an evaluation. ' ^ 



V; Evaluation -as used in this manual refers to 'the , process of s, stematically 
identifying, colrlecting, analyzing, reporting and using information about an - 
educational program dr^ activity. In most cases, people want evaluation infor- 
mation for one of two;^ reasons -to have a bette'tf^nderstanding of a'program 
or to make, decisions k*^ program's future. 

More specifically, people use evaluation information 

-'■■■y'-\-^:-9"'- to'^ understand better, what a program or activity is intended to 
, accomplish,^ w^ it uses and what results it is producing 

m to determine whether a program ^s objectives are important, attain- 
able and relevant to the particular persons involved with the • 
■ program 
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to decide whether the resources for a program are-adequate for 
achieving the results expected.^ ^ - 

to identify areas in which program activities may need to be changed 

or i mproved 

to judge how well a program achieved its objectives, its positive and 
negative impac t and whether it should be continued 

to maintain accountabil ity for effective use of resources 



Decisions: What Type Do You Have to Make? 

Most purposes described in the prior section involve some type of 
decision being made at some administrativeOevel . The following table may 
help you better understand the type of decision you are faced with. 

Daniel Stuff! ebeam and the Phi Delta Kappa National Study Committ^ on 
E/aluation have proposed a useful system for classifying educational 
decisions.*^ Their system classifiis decisions as a function of whether 
they pertain to ends or means and secondly, the relevance of the decision 
to intentions or actualities. - 

Thus, all educaticnal decisions can be "classified by 1) intended ends 

(goals), 2) intended means (procedural " designs) , 3) actual means (procedures 

2 

in use), or 4) actual ends (attainments)" as shown in Figure 1.1 



TYPES OF DECISIONS 



ENDS 



MEANS 



INTENDED 


ACTUAL 


PUNNING DECISIONS 

> 

to determine objectives 


RECYCLING DECISIONS 

to judge and react to 
attainments 


> ^ 

STRUCTURING DECISIONS 
to design procedures 


IMPLEMENTING DECISIONS 

to utilize, control and 
refine procedures 



Figure 1 .1 



T 



Stuffelbeam,j Daniel L., et al. Educational Evaluation and Dec'i;sion Making , 
Itasca, Illinois: F. E. Peacock Publishers, Inc., 1971, p. 80. 
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Figure 1/2, which follows, provides a few examples of each of these 



four types of decisions. Using the program or activity you intend to 
evaluate (or are actually evaluating) write down specific examples of 
decisions that are relevant to your situation and show how evaluation 



findings can be used: An example involving the use of needs assessment 



results has been provided. 



At times the evaluation findings will merely confirm a hunch that pro- 
gram staff felt about a topic and they may feel that the evaluation is 
worthless since "it told me nothing new/' However, one of the functions 
of evaluation is exactly that--to confirm staff perceptions with independent 
' data* In this case, the evaluation findings may simply support the correc- 
tive action that a staff has already taken. 

At other times the findings may conflict with the common opinions of 
the staff. When this occurs, further investigation is warranted to deter- 
mine^'the apparent discrepancies. It may be that the staff members were 
influenced to see what they wanted to see. In other cases, the evaluation 
, ^ instrumentation may not be sensitive enough to detect changes observed by 
staff. Sometimes discrepancies are a function of the environment in which 
data are collected. For example, students are more likely to say they enjoy 
reading history if their history teacher asks them the question than if the 
question was asked on. a,n unsigned questionnaire by an evaluator. 

If an evaluation w<iS not designed to provide unbiased data regarding 
some potential decision alternatives, it 1s only by accident that it may 
later be of use in making decisions. But luck is no substitute for planning! 
Even the be_$t laid plans often, fall short of the mark: It is always possible 
that because of problems, in the instruments, in the design, or in data col- 
lection and analysis that the intended purpose of the evaluation was not 
achieved. For these reasons, the person conducting the evaluation (if other 
than the program decision maker) should identify what the constraints in the 
evaluation have^BeBTr^ai'^4^JiQ_what extent the data should be trusted for making 
decisions. If there are serious reservations about the validity of the data, 
then the evaluation findings should be viewed as highly tentative. If thtJ 
evaluation findings are considered valid, they can then be useful in making* 
choices ^mong alternative actions. "\ 
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PROGRAM DECISION EXAMPLES 



' GENERAL QUESTIONS 

Planning Decisions 

1. Is the prograni addressing sonie 
of the nost cntical needs in 
our district? 

2. Is coMnity input used in the 
planning? 

3. Are our objectives realistic 
for the students to whom 
they apply? 



Structuring Decisio ns 

K What funding sources 'a re ai/ail- 
able to support this prograni? 

What staff skills are required 
for this program? 

What are the best ways to 
recruit studenvs for the 
program? 



2. 



3, 



SPECIFIC QUESTION 



Mhdt do conniunity people in our 
nt?ighborhood see as the niost 
critical educational needs of 
senior high school stuJents? A 
needs assessinent could be conducted 
using personal interviews of 30O 
randomly sanrpled adults from the 
comiiunity. 



Implementing Decisions 



Uhat additional inservice 
training is needed? 

How can students better oianage 
their time so as to complete 
program requirements? 

How can parents be more actively 
involved in the program? 



Recycling Decisions 

1. Have students shown significant 

gain in science? 

*«■ 

2., Have school-community relations 
been improved by the program? 

3. Should project funding be 
continued? 

L Should program goals be changed? 



EVALUATION FINDINGS 



The needs assessment findings indi- 
cated a rank ordering of 12 educa- 
tional goals» wtiich were considered 
as priority for iipprovement aniong 
high school students, and indicated 
that 72 percent of the respondents 
would be willing to support a 10 
percent tax level to pay for a pro- 
gram in the priority area-;. 



USE OF EVALUATION FINDINGS 



Since the cnp:,)unity's ranking of 
educational yOals and priority 
areas for senior high school stu- 
dents was substantially different 
from the survey of high school 
teachers taken last year, the ■ 
superintendent wants to appoint a 
steering coimiittee of teachers, 
parents and conmunity leaders to 
draft out some specific ideas for 
developing into a program pr(^posa 



Figure 1.2 
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Now let's find out what you plan to evaluate and your reasons for doing 
the evaluation. In the box below write in the r^me of the program or activity 
you are now or will be evaluating and the reasons why you are conducting the 
evaluation. If your evaluation is intended for decision making, indicate 
next to each reason whether it is for Planning Decisions (PD), Structuring 
Decisions (SD), Implementing Decisions (ID) or Recycling Decisions (RD). 



Name of One 
Program or Activity 
to be Evaluated 



Reasons for Evaluating 



Type of 
Decision 



Figure 1 . 3 



After identifying your purposes for conducting an evaluation, decide on 
the audiences for the evaluation results. In other words, who will need the 
-evaluation information and for what purposes? Later in this book you will 
use this information for helping tc decide the types of data to collect and 
the best way^ of presenting them to the different audiences. In the box 
in Figure 1.4-, check any groups who will need the evaluation information 
either to be informed or to actually make decisions about improving or con- 
tinuing a program or activity. Be prepared to state how these individuals 
might use the information. • ^ 
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Program or activity 
to be evaluated : 

(check appropriate boxes) 



Persons needing 
the evaluation findings 


to be informed 


to make decisions 


Students 






Teachers 







Program Staff 



Non-Program Staff 

Superintendent 
District School Board 



Parents 



Community Advisory Groups 

Community Members^ 

Funding , Agency 

State Department of Education 

Adopting School Districts 

Others (list) 



Figure 1 .4 
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DESCRIBING A PROGRAM 

OR ACTIVITY TO BE EVALUATED 



DESCRIBING A PROGRAM OR ACTIVITY 
TO BE EVALUATED 




What Should Be Evaluated ? Befo^^^^^^inni ng an 
evaluation, it is important tjo^^^ a^ handle on 
exactly what the program or acvivity is. that is to 



be evaluated. This s^^ion will help you describe 



a program or activi/^ and determine what aspects of 



it are most esse^ial to be evaluated. 



— ' — V 

. Describing or undefttandfng wha.t to evaluate often appears deceptively 
easy- "We. will evaluate the new math program" or "We need to evaluate our 
special education curriculum*" But what actually constitutes the "math 
program'\or the- "special education curriculum?" 

Below are three steps to help you cUri^^y your program "whats." 
Completing each of these steps to some degree is necessary prior to planning 
the evaluation. The "degree" to which each step is completed is, of course, 
up to you. ^ (You know your role and your needs.) Jhese are. the steps involved 
in clarifying what is to be evaluated: 



step 1 
Step 2 

Step 3 



Write a statement which describes why the program 

"or-activity- is-neededr^^^^^^ - 

Write a statement which describes what the pro-- 
gram or activity is -supposed to accomplish. 
(Recall the planning decision step.) 

Write a statement which describes how the program 
or activity is to accomplish what- is intended. 
(Recall the structuring decision step.) 



Figure 2.1 

The information you provide in completing the steps wlllaid in planning 
•conducting and reporting the results. (This type of information 1s generally 
called for in written reports such as project proposals- ) Lei^ us briefly 
discuss each step. 



Step 1 



Write a statement which describe, 
or activity is needed. 



Figure 2.2 

You may already have program documents^vi^^^oposal s which detail this 
iaformation. If so, review these statements and improve them if necessary. 
If not, you may find the informations^^n one or more of the follow >.g forms: 




he program 



1. 
3, 



Needs analysis or assessi^nt (e..g. , a previous evaluation may have 
turned up certain stud^h; needs which your program was intend\^d 
to Tnefet) 

Program rationale^atement 

Rationale section of a legislative mandate 



If this information is not available, write a paragraph that describes 
why the program is needed. Knowing why a program exists prepares us for the 
next step. * ^ ^ . . 



Step 2 



Write a statement which describes what the pro- 
gram or activity is supposed to accomplish. 



Figure 2.3 
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This information may exist in the form of program expectations, goals, 

objectives or Intended outcomes. It may address all or only part of a needs 

statement. The use of goals and objectives is the most common way of express 

ing what the program is supposed to accomplish. Here are brief definitions 

of these terms: . 

• * - ■ ■■*. 

Goals: A general, broad, long-term or Ultimate end for which 

the overall program exists 

Objectives: A specif ic,^measurable outcome of the program or a part 
of the program 

A program goal is generalTy broad and can include specific objectives 
that are derived from it. . For example, a program goal "may be that 'i:students 
wi IT show -an increased proficiency in mathematics," ' ^ - 

Some objective! deal with student outcomes while others deal with 
program processes ■ Here is an example of a student outcome obj^^ctive that 
was derived from the broader goal of increased profic^iency in mathematics.^ 
"Students in the fifth grade- math class will answer correctly at least 12 
out of 15 multiple choice items involving multiplication of two-digit numbers 
in a 20-minute period. " An example of a; process objective is "The. Title I 
Advisory Conmittee will meet at least six times during the school year to 
review program plans and activities and to suggest new activities*" - ^ 



A clearly stated objective identifies: 

• Who wiJ 1. do the action , ^ 

• What the activity is _ 

• Criteria for judging the successful completion of the 
objective, and 

• Conditions under which the activity wiTT be conducted 

Figure 2.4 

Applied to our student outcome objective: 

Who — students in the fifth grade math class are the subjects \ . 
"^ xWhat" -answering multiplication test items correctJy in the activity 

Crivteria^ -l 2 out of 15 items correct .is the criterion lavel 

Condifrlons- Tthe 20--minute time period helps to define the environment 
"X^ in which the activity is to occur - , 



Applied to the process objective: o 
Who— the. Title I Advisory Committee 

Wha t— wi 1 1 meet-to-rev-1 ew-ppoqram -pl ans- -and activities and to suggest 
new activities 

Criteria— at least six meetings; the level of involvement (i.e., not - 
just listen to a description of what Title I is doing but 
actually .review plans and activities and suggest new 
activities) 

Conditions — these actions will occur in committee meetings 
If your program already has clearly stated goals and objectives (both 
outcome and process objectives), go on to the next step. If tJiey do not 
exist, some time will need to be spent developing or refining them. In the 
next section, you will learn how to assess the extent to which your objectives 
are worthwhile implementing. - 

Worthwhile Objectives 

^Usually "goals" are too abstract to measure directly. , Objectives, on 
the other hand, are usually stated so that they can be. measured. It is 
relatively easy to write measurable objectives, but it is not as easy to ' : 
write worthwhile objectives. To help keep the various program objectives 
in perspective, ask the following three questions of each objective. If you 
answer "no" to a question on a given objective, that objective becomes suspect 
and may need revision or elimination. Three questions to ask concerning the : 
worthwh.il eness o"f any .objective are: ' • " 

1. Is this objective clear? For example, ' 

a. Does the objective tell us specifically what is to be measured? 

b. Will the measurement tell us something useful? ' . ^" 

2. Does the objective call .for the performance of an important^ skill 
" --.^ Qr the" implementation of an important process? For example, 

a. Does the student need to know this skill for use outside the*^ 
classroom? 

IN 

b. Is this skill, a tool, which would help" the student at. a later 
date achieve a more important objective? 

c. Is the process consistent with the purpose of the project? 

d. / Is the process directly related to other important outcomes? 



3. Is the objective a challenge and yet achievable^ : For example, 

a. Will the objective cause studentis to have to learn something 
new? 

b. Is the objective realistic for the type of students for whom 
it was developed? 

c. Is it realistic to expect an Advisory Coimittee to suggest 
new program activities? ■ 

Next, we will discuss a method to help determine which objectives are 
most critical to evaluate. 



A Hypptheiical Project 



One of the requirements of the funding agency for a reading project is 
that you report your findings in terms of reading growth resulting from 
special treatment. When the project was approved in your district, the 
superintendent and board indicated that they would install a program as a 
regulan^'part.of the. school system if, and only if, it: 

|lv. Resulted in substantial improvement in reading performance of 

V' students at the end of the period,, and - ' . 

Z: Cost no more than the current reading program. 

Some of the staff members wanted the results to be lasting, that is, 
to .carry over year after year. They also wanted the attitudes of students 
to improve toward: reading. Others wanted to insure that teachers received 
adequate inservice training for the new program and that the program was.,,, 
adequately explained to parents. 

The funding ends * this year , and you have a limited evaluation budget. 
Therefore, you would like to do as much of the evaluation as you can yourself 
using the evaluation specialist as a consultant to help you in ar^as where 
you are inexperienced. 

Objectives as stated for the project included, the followino: 

1. By the end of the school year, the fourth QCC^^^^Eg^^'ni * 
students will have made significant g^^owtW'^^^^^^g , 
as measured by the Woodcock Reading Compr^ef\er!^^^est. 

2. Students will -be able to read a sample newspa^vir art^i^^^ 
correctly in writing four out of six questions dealing with an 3:^^^^^^^^^^^^^^^ 
understanding of facts C9ntained in the sample newspaper article.: 



3. Students wiTl read rnore 1 ibrarybobks on their own initiative than . 
the nontreatment group and select reading more often as a leisure- 
time activity- 

4. Teachers will receive at least 12 hou^s of Inservice training to 
prepare them for the new program. At least 80 percent of the teachers 
receiving this training will indicate on an.unsigned questionnaire at 
the completion of the training that they feel the training was valu* 
able in preparing them for teaching the new program* 

These are four typical objectives; however, if you consider the*.questions 1 
that might be asked about a special reading program, the number becomes end- 
less, e.g., questions about cost., about effects on staff, effects on recrea- 
tional and leisure-time activities of students, etc. 

The importance of rating individual objectives can be judged in light 
of three considerations:.!) legal commitment , 2) the value of the information 
to the decision makers, and 3) logistical conditions , such-as the availability 
of staff , ->time, or money, as shown below. 

1. The agency which funded the program requires a certain set of data. .' 
This is a LEGAL COMMITMENT and must be met. ^ 

2. . The superintendent or school board said this information would be 
" used as a basis for making decisions about the future of the pro- 
gram. This is' a VALUE consideration. ' - 

3. The collection of longitudinal data, would take evaluators three 
years to collect. This is a LOGISTICAL consideration. . 

These categories are organizational headings to help you better examine each 
objective's priority for being evaluated. . ' 

Your next step is to rate each objective in terms of its priority for 
being evaluated. Then, using the ratings as guidel ines, .give the objective 
an overall rating according to- its priority. Mathematical aver^es across 
the scale^' are not appropriate* since an area that^ is required by law to be 
evaluated will automatically receive top priority even if ther^e may be some 
logistical problems in collecting the information. 

If your program or activity has only several objectiv^es, it is probably 
not necessary to utilize the forma.l procedure of filling out a form such as 
that on' the following page. However, this process is useful when you have 
more objectives or evaluation questions than you feel your resources will 
allow for .eval uating. • With your goals and objectives prepared, you are 
ready for the final step in the program description. 
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OBJECTIVE RATI^IG FORM 



Objective to be. /rated 



Place an "X" in the appropriate place dr\ the 'line 
to the right after considering each- category as 
it relates to your particUlan^ objective. 

Legal Commitment 

1.,. Requirements in federal* state or local 
guidel ines. 



Top 
Priori ty 



Moderate 
P riority 



Low 
Priority 



Value to Decision Maker:3 



2. Importance to .the future of the program. 

3. importance to decisioniirnakers. or curriculum 
imprjvers'^t / . ^ ' 

Logistics ■ . 

4. Extent to which the objective has already- 
been fully implemented. 

"'■ ' . ' • ■ , ' c 

5. Extent to which information regarding this ■ 
i tern -has al ceady been collected. (If so, do 
.we need to re-evaluate. it?) 

6. ^ AvailabiTity of . sufficient funds to carry 

ou.tJian, evaluation of this objective. 

7. Sufficient time available to evaluate it. 

8. Availability of .the expertise needed to 
evaluate this objective. 

9/ Presence of supportive processes leading to, 
.an fOUtcome objective. / 

10. Others (list) 



Facllita tes 
Evaluation 



TT 



T" 



^ Hinders 
Evaluation 



Based on the above profile, what priority rating would you give to the evaluation of 
this objective? (Circle one) ^. 



Top 
Priority 



Moderate 
Priority 



' Low * 
Priority 



Not Appropriate 
to Evaluate. 



ERIC 



Figure 2.5 
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Step 3 



Write a statement which describes how the program 
or activity is^xo accomplish what is intended. 



7 F/igure 2.6 

This step calls for: program jimplementstion plans. The following key 
words may help you include most, of the relevant facts in your description, 
- / Program: : | * 

5. Proc;ram Personnel | 

... ■ , ' ' ■ j 

6- Resources to be Used 

7. ^ Time Schedule ■ " ■/ 

8, Costs 



1. Selection of Participants I 

2. Participants' ' Charactefilt.icsi 

3. Instructional Activities \ 

4. Management Aqtivities 1 



After having completed the three steps in this^^section of the manual, 
you should now have a clear description of why the program being evaluated 
exists, what it is supposed to accomplish and how it is supposed ^to do it. 
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ESTABLISHING EVALUATION 
GUIDELINES 
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ESTABLISHING EVALUATION GUIDELINES 



What Fact o rs Influence the Way This Evaluation 
Win Be Conducted ? Many. Guidelines from fund- 
ing agencies; di.strict policies about parental 
permission for students to be tested, maintain- 
ing confidentiality of data, and approval of 
e^valuation inst/uments prior to their use; 
availability of district resources in conduc- 
ting an eval ua tion--al.l are important factors 
governing an evaluation. 
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Funding Agency i 

When a district is receiving state or federal funds to operate a 
particular program. It is essential for people involved in the evaluation 
to be familiar with any evaluation guidelines or requirements prepared by 
the funding agency. Fqr example, a district receiving federal Title I funds 
under the Elementary apd Secondary Education Act has certain types of ^eval- 
uation data that must be collected and reported. If school staff are not 
sure of these requirements, a district or state Title I coordinator should 
be contacted. The funding agency may also specify constraints in the 
collection of the evaluation data such as those related to HEW guidelines- 
for protection of human subjects . 

District Policies 

Even if no outside funding is used to operate a program, school admin- 
istrators should check with the district to determine any local policies o-r 
laws related to program evaluation. For^example, some district^ have a 
policy regarding the release of group evaluation data. In some cases, all 
evaluation reports are presented to the superintendent, who is then respon- 
sible for disseminating them to the Board of Education and the public. 
Recent legislation regarding parental knowledge and approval of certain 
types of student testing and the disposition of evaluation infdrmation is 
also an area with which school administrators need to be familiar. 

Local Concerns 

In addition to district pol icies and laws related to evalu<?ition, a 
local school may wish to establish some basic guidelines related to evalua- 
tion. For . example, some schools require that- evaluation instruments be 
reviewed and approved by the Community Advisory Committee prior to their 
use* This review can be especially important in detecting sensitive items 
that would be considered offensive or an invasion of privacy in certain 
communities- 
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Evaluation lEthics 

As can be seen in citing one example of local concern/ the issue of 
ethics is critical to the effectiveness 9f the evaluation. A checklist, 
see Figure 3J, was developed to help you in planning and implementing 
your evaluation. Being aware of potential problem areas at this early 
point in the evaluation may help you avoid these problems later. Read 
through the form and refer: to it as you begin to plan and develop 'future- 
aspects in the evaluation. 

Identify Resources- 

Evaluation guidelines .can also^specify the resources available for 
conducting- an evaluation. Before preparing an evaluation plan, it is use- 
ful to estimate th_e budget available for evaluation and to identify 
personnel and resources likely to be available for the evaluation. These 
considerations often influence whether ah evaluation will be conducted by 
school staff, evaluators within the district or external evaluators. 
Available resources within a district can also affect the tasks to be 
performed by an external evaluator. For example, if an external evaluation 
is being contracted and the local district maintains an efficient print 
shop, it is.-probably more coi c-eff ecti ve for the external evaluator to 
prepare a camera-ready copy of data collection instruments and evaluation 
reports for printing .in the district facilities. 

District Assistance 

In considering personnel within the district who could assist in an 
evaluation, it is often useful to look beyond persons trained ^particularly 
in re,search and evaluation. For example, a district's director of curricu- 
lum and 'instruction might be an excellent person to evaluate a sample of 
students* individualized projects. Teacher aides can sometimes be trained 
to be effective data collectors. 
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CHECKLIST ^FOR INSURING THE ETHICS OF Af| EVALUATION 






Evaluation PMn 


A Problem 


No Problem 


1. 


Has the evaluation plan been rr/iewed to insure that the 
treatment of students in a program or comparison group is 
fair? - 




2. 


Has parental permission for student testing been obtained, 
if necessary? 


- 






Are all of the data to be collected directly relevant to 
the evaluation needs of a program? 








Evaluation Instruments 






4. 


Are the evaluation instruments , or thetr assumptions offensive, 
in the opinions of educators or conmunity members? 






5. 


Are the <Jirections for completing evaluation instruments 
clear enough for -.the respondent to know the purpose or use 
of the instrument? ■ 






6. 


Are any of the items on an evaluation Instrument likely to 
cast doubt on the respondents' opinions of themselves, 
their family, heritage, culture or values? 






7. 


Are any of the items likely to maintain or promote racifll 
or sex role stereotypes? 








Data Collection Codinc, and Storage 






8. 


Are people given the.option of not completing an Instrument 
or any item on it they find personally offensive? 






' 9. 


If interviews are to be taped, is it done with the full 
knowledge and consent of the subjects? 






10. 


Are data collection personnel trained to keep sensitive 
data confidential and to properly administer the assessment? 






11. 


Are students* names kept separate from other sensitive data 
when recorded on keypunch cards, tapes or printouts? 






12. 


Are raw da^a such as completed questionnaires stored 
securely and then destroyed when no longer required? 








Interpretation and Reporting of Information 






13. 


Are student test results interpreted by a qualified person 
in such a way as to be properly understood by the student 
or parents? 


— 




14. 


Is the infonnation repor^ted on individual students ade- . 
qudtely rel iable for use' on an i ndi vi dual bas is ? 






15. 


Is evaluation information released to educators or the 
public in such a way as to avoid harm to an individual 
or group? 






16. 


Does the evaluation report cover both the positive and 
negative findings when present? 






17. 


Are limitations irr the-evaluation design>. instruments 
or analyses spyecified in the report to keep decision 
makers from reaching unwarranted conclusions? 








Other Areas of Importance 








(List) 

■'"^ /' 







Figure 3.1 
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Use of an Externa! Evaluation 

In the case where avdistrict does not have a skilled evaluator available 
within the district and funds are available, it may wish to contract exter- 
nally for all or part of the evaluation. An externally conducted evaluation 
generally has the advantages of being more independent, providing findings 
considered to be more credible by the evaluation audiences and conserves 
district staff time. Some common limitations of extcirnally conducted eval- 
uations are that: the evaluation personnel do not always understand the 
details of the program or their rationale; it is often more expensive; 
external evaluators may sometimes be viewed with suspicion if not distrust; 
and the, evaluation findings may not be as useful in leading to specific 
program modifications. 

When selecting external evaluation by individuals or agencies, you 
should consider the following factors: 

• their training in evaluation methodology 

t ;past experiences in evaluating similar programs 

• skill in areas of special need, such as test de^^ 

t data analysis capabi*^ 1 ties (if these are a relevant part of their 
task) 

• skills in relatinr - tudents, staff and other groups, such as 
parents or community volunteers 

. • availability and geographic proximity to the district 

• lijeir cost in relation to the services they agree to provide 

If a decision is reached to contract with an external evaluator, a 

A' 

district administrator needs to negotiate a written agreement delineating 
the responsibilities of the district and of the external evaluator for the 
design, collection, analysis and reporting of data. Time spent in prepar- 
ing such an agreement is more than compensated for by the elimination of 
possible misunderstandings and conflicts at a later period. 

An outline of sample items to be addressed in the agreement is found 
in Figure 3.2. The administrator should keep in mind that the allocation 
of responsibil it:ies as shown on the following two pa'ges is only an illus- 
tration, You and youi- evaluator may want to modify it to fit your paj^icu- 
lar situation. / 
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■ EVALUATION RESPONSIBILITIES OF THE DISTRlCr 
AND OF AN EXTERNAL E VALUATOR 


TASK 


DISTRICT 


EXTERNAL EVALUATOR 


Coordination 


Designate a person to coordinate 
the evaluation responsibi 1 1 ties ' 
for the. program 


Designate an evaluation person 
as a primary contact person 
for working with this program 


Evaluation Plan 


Review the general evaluation 
plan and revise as necessary to 
fit the project. Return the 
revised plan to the external 
evaluator ...-^ 


U ' Prepare a general evaluation 
plan in cooperation with the 
project staff 

2, Revise and .approve the 
district's revised evaluation 
^ plan r ' 


Instrumentation . 


1. Reproduce required copies 
of all evaluation instruiDents 


1. Prepare a draft copy of all 
instruments to be used 




2. Order required copies of 
standardized instruments^ and 
answer sheets 

3. Develop any local monitoring 
or evaluation instruments 

4. Obtain a review and approval 
by the district and^school 

. officials for use of each pro- 
posed evaluation instrument 


2. Provide the district with a 
specimen set of standardized 
insLrumenus to oc useu toqcbncr 
with cost Information and an 
order blank ^ 

3. Review any district- 
developed instruments if 
requested by the district 


Data Collection 


1. Schedule and administer all 
evaluation instruments identi- . 
fled in the evaluation plan 


- 1. Provide the district with a 
schedule and design for data 
collection. 




2. Collect and code file. data 
specified in the plan " 

3. Code responses to all in 
instruments where needed 

4. Mail a duplicate copy of 
all code sheets to the external 
evaluator for computer process- 
ing 


2. Provide written directions 
fnv* ;iHmi ni cf pri no nnn^tandardi zed 
evaluation instruments 

3, Prepare coinmon codes and 
coding directions for all answer 
sheets and data collection forms 

•>"» 


. Data Analysis 


Identify if there is any special 
data analysis the district 
would like to have. run that has 
not already been included in 
the evaluation' plan - 


1. Verify the correct scoring 
and/or codinq of all instruments 

2* Keypunch the data 

3. Provide scoring services 

4. Analyze the data 


Reporting 

<• 


1. Identify the'informatlon 
needs of the people in the 
district if they have changed 
since' the evaluation plan was 
prepared 

2. ' Review the draft evaluation 
report for any factual errors' 
or misrepresentations 

3. Print the required number/ 
of evaluation reports and 

■'abstracts 


" 1. Prepare a draft copy of the 
evaluation report and give it to 
the district evaluation coordi- 
nator for review 

? Pronnrp ;i f 1 rtpi 1 carft^ra^readv 

^ ( r I C r c a , 1 1 ifa 1 bail Id u « yuj 

copy of the evaluation report 

3. Prepare a caRiera-ready copy 
of an evaluation report abstract 



Figure 3.2 



Estimating Evaluation Costs and Personnel Requirements 

: The costs and external personnelrequired in conducting an evaluation 
depend .on a number of factors such as the type of evaluation to be conducted, 
the amount of work done by the Internal staff, the size. of the project, and 
travel and fees of the ^valuator. Evaluation costs can usually be grcuped 
under the following categories: personnel costs, travel, purchasing • - 
evaluation instruments, data collection, data coding and scoring, data 
maintenance, data analysis, report writing, and presentation of findir.gs. 

r • ... 

A rule of thumb used by some administrators in proposal writing is 
to estimate the evaluation costs to/be between five and ten percent of the 
total 'program budget. Larger projects (for example, those for over $150,000) 
often can have an adequate evaluation for closer to five percent of the total 
budgeit while smaller projects (those under $50,000) may need more than ten 
percent since many functions such as instrument development are required 
regardless of the number of students tested. 

Now that we have reviewed some of the important jconsi derations in 
establishing evaluation guidelines, turn to the next page and check the 
appropricte columns in the box. to see whether you have accounted for these 
important considerations that influence the way your evaluation will be 
conducted. 

'-• • • 



To 



REVIEW OR ESTABLISH EVALUATION GUIDELINES 



Guideline Considerations ■ 



I Have 
Reviewed/ 
Established 
These . 



\ These Have ^ 
Beeri Applied 

to the 
Evaluation 

• Guidelines 



More 
Work 
Needed 



1. Evaluation guidelines' or requirements of the, 
'funding agency (if applicable) 



2. Legislation regarding' parental permission, 
confidential tty of data and policy concerning 
release of student records , ■ 

3.. Distnctpolicies concerning evaluation, 

4. School level policies concerning evaluation ■ 

5,, Evaluation budget available , 

6. District/school personnel available to work on 
the evaluation 

/.. Policy regarding the use of the district's . 
' computer, -printing and other facilities \ 

8. External evaluator's roles defined (if applicable) 

Other. areas (list). 
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PREPARING AN EVALUATION PLAN 
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What Should Be Covered in an Evaluation Plan ? An 
adequate evaluation plan requires that certain 
questions be .answered beforeyou begin your data 
collection. In .this section you will fill out, 
step by stepj the elements of a complete pi an . 
You will., learn, appropriate cri teria to a)pplj^^ to 
your evaluation plan or to those deyeloped by 
others. A well developed evaluation pla^n can 
save you time and headache in carrying out an 
evaluation and help to insure that th^e ^'data you 
collect and report win be useful for decision 
making. The. eval ua t i on p 1 an should be formu- 
lated before or at the outset of a program's 
impl ementation . 



E.lgHt^'&^en)efttft:5#f comprehensive:! evaluation plan are shown in 

Figure"4.1;&yi|(Gi^ discussed Oif^^l^jEd^ lowing pages. 

. " Evaluation Plan Outline 

r ■ ■ - ■■ ■ 



.Step 


Question | 


Objectives/Issues 


^ 

What questions are' being 
asked? 


2. Information Requirements 


WHat information is needed to 
answer- the questions? 


3. Information Source 


Fronrvwhom can the necessary 
information be secured? 


4. Instruments 


What can be used to find, the 
answer?. 


5. Design 


Who will complete the instru- 
ments and what comparisons 
may be made? . 


6. Time Schedule 


When will the information be 
collected, analyzed and 
reported? 


7. Analysis 


What do we do with the data? 


a . ^Report 


Who needs to know about it? • 



Figure 4.1 



Step 1. Objectives/issues 

What Questions Are Being Asked ? In Section 2 of this manual you 
learned how to refine program objectives and to identify a priority of 
which objectives or questions are most important. for you to evaluate in 
a given period. 

■ ■ ■ ■ ■ 3y 



nrHe^:&J'ec1:1ves you are planning to-evalua-te-or-the-q 
arising from those objectives which you hope to answer may 'deal with 
infonnation concerning areas sych as student outcomes , efficiency of your 
staff, community attitude, or availability of extra movie projectors, the 
evaluation question can be generated by discussions of the project staff 
and the', evaluator early in the evaluatiorfTDrocess. However, if the evalua- 
tion Is to be a useful tool for program improvement it is important to be 
able.:^^^^^ hew questions as they arise throughout the yeaf*. The 

sairte-^'type of plan described below can be-developed to help you gain the 
most useful information about each objective/questions For an example, 
>net:us^use the objective stated in .Figure 4.2. 

Eval uation Plan 



1 , Objective 



By the end oi- the school year, the fourth 
grade progfiam students will have made a 
significant growth .in reading as measured 
by the Woodcock Reading Test. 



Figure 4*2 - * 

This objective deals with student outcomes • For an example of how 
process objective's can be treated 'in an evaluation plan, see pages 39 arid 40, 

Once you have 'each objective or question recorded on a form, you are 
ready for Step 2. 

Step 2, Information Requirements 



What Information Is Needed to Answer the Questions? An information^^ 
requirement .is the information needed to adequately evaluate a particular 
objective. In this section we will use the objective stated in the box 
above in building the elements of an evaluation plan. ^ " 



Evaluation Plan 



1 • Objective 
» • 


By the end of. the school year, the fourth 
grade program students wil 1 have made a 
significant growth in reading as measured 
by the Woodcock Reading Test. 


2.. InforirotiorV 
Requirements 





Figure 4,3 



"The infQ|prnation you feel is required should be stated in the space 
indicated. You can compare your answer with that shown in Figure 4*4* 

Step 3. Information Source 

Where and How Can^the Necessary Information Be Secured? Sources of 
information are the individuals, agencies, or records from which the . 
required information can be gathered. ^ 

Here are soree^ questions you might ask to help you determine the 

1 " 

appropriate source. 

1. Which sources have the necessary information? 

2. How much information doeS] each source have? 

3. How val i^ and reliable is the 'information in each case? 

4. Will gathering this information from the source be too burdensome 
for the source? (Collecting too much information from a single 
source may cause problems in attitude.or cooperation.) 

5. Which is the most efficient source? 

6. Will more than one source be needed to compare results for 
accuracy? - 

After answering these questions, fill in th^ information in the appro 

priate space on the form below^ You can compare your answer with that in 

Figure. 4.5. ^ ' 



Adapted from "Evaluation and Educational Decision Making," A Publication 
of the Department of Health, Education and Welfare, 1975, p, 42. 
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Evaluation Plan 



1. Objective 


By the end of the school year, the fourth 
grade program students will have made 
significant growth in reading as measured 
by the Woodcock Reading Test. 


2. Information 
Requirements 


The reading scores for the fourth grade 
students administered at the beginning and. 
end of the school year and the test norms./ 


3, Information 
Source 




Figure 4,4 ! 
Evaluation Plan 

> 


1, Objective 


By the end of the school year» the fourth 
grade program students will have made 
significant growth in reading as measured 
by the Woodcock Reading Test. 


2. Information 
Requirements 


The reading scores for the fourth grade 
students administered at the beginning and 
end of the school year and the test norms. 


3, Information 
Source 


The fourth grade students. 



Figure 4.5 



You are now ready to go on to Step 4. 



X ■ 

2 : 

A norm is an empirically derived distribution of scores on a test for 
students around the country at one or various grade levels. 
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step 4. instruments 

What Can Be Used to Find the Answer ? Instruments are the tools or 
means you use to collect information* Instrumentation, then, is the process 
of selecting or developing the instruments you will need There are two 
major types of instruments. The first type is commerciaUy developed. 
Examples are standardized tests you may purchase. Some of these are norm 
referenced while others are criterion referenced (related to prespecified 
curriculum tasks to be performed, e.g., addition of fractions). Here are a 
few points to keep in mind when using standardized tests. 

• They are usually easy to administer, score and interpret 

% Information concerning reliability and validity is generally 
available in the test publisher's technical manual 

• Prepared tests may not measure exactly what you wish to measure 
(but don't change your objectives to meet the instrument) 

The second type of evaluation instrument is often called "locally 

prepared" instruments. These are developed specifically to collect data 

for a given evaluation plan. You may be able to develop these yourself, 

depending on the nature of the information requirements or you may ask a 

trained evaluator for assistance. Below are a few types of commonly used 

locally prepared instruments/ 

1. Tests (draw on standardized tests for ideas but remember their 
testis are copyrighted; also contact the National Assessment of 
Educational Progress for samples of criterion referenced items) 

2. Questionnaires (self-reporting forms) 

3- Observation scales (generally for observing classrooms) 

4. Interview questions 

5. Document review summaries (analysis of School Board minutes) 

6. Staff reporting forms (rating sheets or narrative statements) 

Because the time and resources available for an evaluation are usually 

limited^ it is wise to consider a variety of potential evaluation instru- 
3 

ments and then to analyze the alternatives and to select the minimal number 
needed to accnrnolish the job. Factors to consider in determining priorities 



The reader may wish to read the section on "data sources" in S. Anderson^ 
Encyclopedia of Educational Evaluation , Isaac and Michael, Handbook in 
Research and Evaluation , WortherTand Sanders, Ediicati ona 1 Eval uati on : 
Theory and Practice or Owens, Haenn and Fehren^bacher "The Use of Multiple. 
Strategies in Evaluating an Experience-Based Career Education Program/' 
A complete citation of each of these is located In Section 7 of the manual. 



among potential instruments of equal val idity ^include: 

1 . Costs 

2. Timing 

3. Credibility of findings 

4. Degree of obtrusi veness 

5. Amount of coordination needed 

6. Efficiency " 

Cost 

Cost factors to be considered include the cost of purchasing or 
developing, an instrument as well as the cost to administer, score and analyze 
it. The amount of time involved of students,* staff or community resource 
persons if often overlooked but should be taken into consideration in deter- 
mining costs. 

Timing 

The timing of data collection is another crucial factor. Several 
elements of timing need to be- considered including: 

1. The deadline when the information is needed 

2. The length of time it would take to plan, collect and analyze data 

3. The most appropriate time in the developmental cycle of a project 
for collecting certain data 

Credibility 

Credibility of the findings is something that is often overlooked until 
an evaluation is completed and the data reported. Some audiences such as 
educational researchers may be impressed with "hard data" such as a multi- 
variate analysis of scores from a standardized test. Other audiences, "such 
as parents, may be more informed by well written student case studies. In 
addition, credibility of the evaluation findings can be enhanced when- several 
evaluation instruments produce results that reconfirm or support what was 
found through the use of a single instrument. 



Degree of Obtrusiveness 

Another important factor to consider in selecting evaluation instruments 
is the extent to which they will be obtrusive and perhaps clash with the 
mission and activities of the project. The evaluation of a sample of already 
completed student written reports provides an excellent insight into student 
learning outcomes without requiring additional time of students. On the 
other hand it may be unreasonable and i^nappropriate to administer a th^Jie 
or four hour basic skills test battery twice a year if the students already 
have a dislike for test taking. 

Coordination 

A point often overlooked in selecting appropriate ways to obtain 
needed information is the amount and type of coordination required. This 
coordination includes the amount and type of interference that may accrue 
to students, staff and others such as participating community people. It 
also includes the coordination of persons or agencies to be involved in the 
data collection for each instrument. 

Efficiency 

Although we have mentioned that credibility is enhanced when several 
evaluation approaches related to the same issue produce consistent findings, 
one has to weigh to trade off between a redundancy of information collected 
on a few issues to establish the reliability of the findings versus the 
expense of foregoing the collection of unique information about additional 
issues.' Always check to determine if useful data are currently available 
from other processes such as a district-wide testing program. 

In our example the instrument is clearly identified in the objective* 
In many cases it will not be so easy- This information should be entered 
on the form for Step 4. 
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Evaluation Plan ^ \/ 



1. Objective. 


By the end of the school year,^the fourth 
grade program students will have made 
significant growth in reading as measured 
by the Woodcock Reading Test. 


2. information 
Requirements 


The reading scores for the fourth grade 
students administered at the beginning and 

ClIU UT Uflc: .bCilUU 1 jfCui ailU Lilti CCO U IIUi NIO * 


i ilT vrillct 1 Ui 1 

Source 


Thp ^niiv'hh nK*;?rlp ^1"iiHpn^<j 


4. Instruments * 


r 



Figure 4.6 



Step 5. Design 

Who Will Complete the Instruments and What Comparison May Be Made? 
An evaluation design is a plan or strategy for collecting and analyzing 
information about a program. It also identifies the comparisons or criteria 
to be used in judging the program's success. 

On the next page are a sample of questions you might need to address in 
your evaluation. Under "Design" are listed the designs, methods or types of 
strategies most appropriate to answer those particular questions. Find the 
question which best describes your situation and note the appropriate 
strategy. If the strategy is one with^which you are unfamiliar or which you 
know is beyond the expertise of the project staff, you should consult an^^ 
evaluation specialist. Some references are given in Section 7. .Your ques- 
tions may re<:ju.ire entirely different designs than those shown on page 32. 



Sample Questions . Corresponding Desirjns 



We know how wel 1 the fifth 
araders ran SDell from 03St 
testing and experiences (base- 
'line data). Will this new 
'spelling program improve their 
spelling over' what we expect? 

(ST • ^ • 


Administer- at the end of the 
school vear the same soellina 
test used in the past to this 
year's fifth graders arid com- 
pare the results with the base- 
line data using available 
norms. 


) ■ 

We need to know at what level 
entering sophomores are in 
.their math skills and whether 
this new program improves 
those skills. 


Administer a math test to the 
$ophomores at the beginning 
and end of the school year. and 
compare their results* 


How effective is the new 
fourth grade reading program 
as compared with the? one we 
have been using? 


Randomly assign half of the 
fourth grade students in a 
school to one reading group 
using the new program and the 
other half to a group using 

thp nl Hpr* nr^nnr'am Hr^v/P thp 

UIIC \J 1 UCl Ul wMi Ulll. IIUV^ UlIC 

same, teacher teach both groups.' 
Select a reading test equally 
fair to both programs and com- 
pare the posttest results of 
each group. 


Do the seniors in our new 
career education program 
make greater- progress in job 

infprvipw skills than'stu— 

dents in our vocational ' 
education. program? . 


Develop a simulated job inter- 
view and randomly sample per- 
haps 20 students from each of 

thp two ornijn^ fn nartirinatp 

in the simulation >at the begin- 
ning and end of the school year. 
Have the simulation scoffed and 
compare the increase in scores 
made by the two groups. 



Figure 4.7 " 
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Enter the appropriate design on the Evaluation Plan form. Figure 4.8 



Eval uation Plan 



1, Objective 


By the end of the school year, the fourth 
grade program students wHl have made 
significant growth in reading as measured 
bv the Woodcock Readina Test. 


2. Information 
Requirements 


The reading scores for the fourth. grade 
students .admi nistered at the beainnina and 
end of the school year and the test norms. 


3. Information 
Source 


The fourth grade students. 


4. Instrumsnts 


California Achievement Test, Form C 


5. Design 





Figure 4.8 

If your answer involved comparison of the pretest and posttest results, 
which the objective calls for, you are ready to go on to ^ Step 6. 

Step 6. Time Schedule 

When Will the Information Be Collected, Analyzed and Reported? The 
first question to ask is when will each instrument be administered. Some-^ 
times an instrument such as a questionnaire will be administered only a 
single time. If an interview or a test is administered twice (pre-post) 
or three times, these times should also be recorded. Other data often 
called "process data" refer to ongoing data collected periodically through- 
out a school year such as student attendance data for each grading period. . 
Fill in appropriate times in the box on the next page. 



Evaluation Plan 



1. Objective 


By the end of the school year, the fourth 
grade program students will have made 
significant growth in reading, as measured 
by the Woodcock Reading Test. 


2. Information 
Requirements 


The reading scores for the fourth grade 
students administered at the beginning and 
end of the school year and the test norms. 


3. Information 
Source 


The fourth grade students. 


4. Instruments 


Woodcock Reading Test. . 


5, ' Design 


Pretest- pes ttest. 


6. Time Schedule 


Pretest— September 15-18 
Posttest— May 12-16 



Figure 4.9 

Now that you have entered the time schedule, you are ready to' go on 
to Step 7. ' 

Step 7. Analysis 

What Do We Do With the Data? Data analysis is the process of making 
useful information out of raw data. 

This can be as simple as counting hbw many people liked and disliked 
certain textbook, to some very complex statistical manipulations. 11" you 
have little or no statistical background and the analysis requires more th 
percentages and graphs, it is best to consult an evaluator. The-analysis 
technique should be listed on the form below. 



Evaluation Plan 



\ 



1 . Objective 


By the end of the school year, the fourth 
grade program students will have made 
significant growth in reading as measured 
by the Woodcock Reading Test. 


2. Information 

Requiremenls-^ — 


The reading_scojc:eS---f-ar-'tfTe'^fourtfi grade 
— s-ttrdHrvtTl[clministered at the beginning and 
end of the school year and the test norms. 


3. Information 
Source 


The fourth grade students. 


4. Instruments 


Woodcock Reading Test. 


5. Design 


Pretest-posttest. 


6. Time Schedule 


Pretest— September 15-18 
Posttest— May 12-16 


7. Analysis ^ 





Figure 4.10 



Although at this point you may not be familiar with the many alterna- 
tives of data analysis, the key features" of some of the most commonly used, 
statistics are explained on the following pages to give you some orientation. 
Keep in mind whether you will have to be performing the calculations by hand, 
with the assistance of a programmed calculator or will have access to a 
computer and avai lable statistical packages. 



KEY FEATURES OF SOME COMMONLY USED ANALYTICAL TECHNIQUES 



Analytical^ 
. Technique 
and its 
Symbol ' 


N 3 

Key Elements 


Example * 


CHI SQUARE 

■t 


Used to test the si gnif ic-ance of 
differences between: (1) observed 
frequencies and expected frequen- 
cies; or (2) two sets of observed 
frequencies. 


Is there a significantly larger 
proportion of boys than qirls who 
applied for entry into avnew science 
program? 


t TEST : . 


Used to test whether the differ- 
ences between two means (group ^ 
averages ) is significant . It can 
measure the differehce between 
two groups or between the. pre- 
and posttest scores of the same 
^roup. 


Did the students in the home eco- 
nomics course show significant 
growth on the HE203 test between 
the beginning and end of the 
semester? 

Did students in the. Head Start pro- 
gram score significantly higher 
on tl^e Peabody Test than youngsters 
who remained at home? 


ANALYSIS OF 
VARIANCE 

(ANOVA) 


Used to test the significance of 
differences among more than two 
variables or more than two groups. 


Which of three methods of teaching 
word recognition is most effective 
for slow learners. 


Mill TTDI C 

MULTIPLE 
ANALYSIS OF 
\VARIANCE 

(\l^NOVA) 

\ 


Used to test the significance of 
differences between two or more 
groups simultaneously on more than 
one variable/ 


Do students in the experimental 
■ " school program score signif i- 
'V*: 'l^f/ higher in basic skills (as 
mecisured by separate scores in 
reading, arithmetic and language 
expression) than those in the 
traditional p ro g ra m ? 


ANALYSISVOF 
COVARIANCE 

(ANCOVA) 


Used to Test the significance of 
differences between two groups 
when the two groups are not con- 
sidered equal at. pretest time. 


Do students ^in the work experience 
program make greater gains in 
career maturity than those in the 
vocational education class when 
students' IQ is accounted for? 


MULTIPLE 
REGRESSION 


Combination of several predictive 
measures to predict achievement 
or other outcomes. 

^ 1 


Do students' age> sex and pretest 
scores on an achievement test sig- 
nificantly predict the-performance 
outcomes of the program* as measured 
by posttest scores? 



Figure- 4. 11 



[ Because the sample objective calls for measuring significant growth 
over the year, the analysis needs to determine whether the differences in 
the group's pretest and posttest scores are statistically significant or 
whether they could have occurred by chance alone. As indicated in 
Figure 4,11, a t test or analysis of variance would be appropriate to use, 
Because statistical significance is based on a mathematical formula that 
incTudfes the number of students tested, it is possible that if a large 
number of children were tested, a difference between the group's pretest 
and posttest averages of only one or two points could be stati stically 
significant. However, it is also important to ask the question as to 
whether the differences were educationally significant. . In this case a "gain 
of only one or two points may indicate very little actual growth- 

While the chart in Figure 4.11 deals with statistical techniques for 
analysis of data, it is- important to remember that simple- descriptive 
techniques- such as graphs, bar charts and scattergrams are often effective 
when communicating the evaluation findings to persons not having a statis- 
tical background- 
Step 8. Reports 



Who Needs to Know About It? Reports are ways-of conveying information 
to others- The types of reports (e-g- . mid-year) and the recipients of the 
reports (e.g-, the superintendent) should be entered on the form in the 
space provided- You may wish to go back to the explanation of this hypo- 
thetical project on page 11 and review which individuals and/or agencies 
wanted the information about the evaluation findings- Insert on the next 
page the individuals or groups to receive the reports- 
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Evaluation Plan 



1. Objective. 


By the end of the school year, the fourth 
grade program students will have made 
significant growth in reading as measured 
by the Woodcock Reading Test. . 


2. Information : 
Requi rements 


The reading scores for the fourth grade 
students administered at the beginning and 
end of the school year and the test norms. 


3 . intorina ti on 
Source 


Tha^ourth "grade students-,. 


4. Instruments 


Woodcock Reading Test. 

i 


5. , Desi gn 


Pretest-posttest . 


6. Time Schedule 


Pretest--September 15-18 
. rosttest--May 12-16 


/ . Ana 1 ys i s 


uompuue a l uesu uo aeicermine it a sigmti- 
cant growth (at the, .05 level) occurred. 


8. Report 


. 1 



Figure 4.12 

Important audiences to receive the final report are the funding agency 
together with the superintendent and school board,. School protocol requires 
that ahyHnformatipn going to the board of education be submitted to the 
superintendent, who in turn sends the selected infprmation required to the 
board together with recommendations, if any. Evaluators should consult 
with the superintendent or administrative representative regarding these 
matters. The superintendent may request that you make .an .oral presentation 
to the board, parent groups, or other audiences. In addition, it may be ^ 
:Usef'ul to prepare a separate brief summary of the evaluation, findings. 



Evaluation Plan 
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1 . ~ Objective 


By the end of the school year, the fourth 
grade program students will have made 
significant growth in reading as measured ' 
by the Woodcock Reading Test. 


2. Infonnation 
Requirements 
• **" 


„ . - , « 

the reading scores for. the fourth grade 
students administer'ed at the beginning ^ind 
end of the school year and the test norms. 


3. Information 
Source 


o- ■ 

The fourth grade students. 


4. Instruments 


Woodcock heading Test. V 


5, Design ^ ^ 


n . . ■ ' ' ' • 

Pretest-posttest. 

■ i 


6, Time Schedule 


Pretest— September 15-18 

Posttest— May 12-16 .^.^.t... 


7. Analysis 


Compute a t test to determine if a signifi- 
cant growth occurred. The test publisher's 
norms may also be useful to provide a 
descriptive level, in terms of grade 
equivalent scores j of the group's pretest 
and posttest means. 


8. Report ' 

/ 


" ■ • ■ •> 
Final report submitted to* superintendent ; 
and funding agency. 


/ 

f ■ 


Figure 4.1 3 . ' 



Process Objective Example 

Pages 24 to 38 have illustrated how eight steps can be used for preparing 
an evaluation plan related to a student outcome objective. A comprehensive 
evaluation plan may also address a program's impact on the community and. 
other agencies. It may include a look at process objectives that are •'"'"T"^. 



• 40 



instrumental in achieving the outcome objectives. An example of a process 

objective is shown .in Figure 4.14 together with steps useful i-n evaluating 

- / ■ ■ 



it. 



1. Objective 


Teachers in the Webster reading orogram will participate in a four-day 
training workshop to learn the purposes of the curHcuTum and how to 

effectlv^lv imDlpnwnf if 


2. Information 
Requirements 


!• A copy of the training workshop deslcn (includina purposes, 
expected outcomes and teaching strategles). 

2. Names of the workshop staff and of the teachers scheduled to attend* 


3. Information 
Source 


1. The workshop staff 

2. Workshop participants 

3. Wor>ic^hop observer 


4, Instruments 


\ 

1. Observation schedule 

2. Pretest and posttest for participants 

3. Workshop garticipant questionnaire ' 

4. Attendance sheet , : 


5. Design 


• 1.^ Pre and post administration of the knowledge test. 

Z, The evaluator will attend the session and complete the observation 
schedule. 

3. Participants will complete the questionnaire and the daily atten- 
dance sheets. 

4. Workshop staff win be interviewed after the workshop. 


6. Time Schedule 


1. Pretest to be administered as the first activity. 

2. Posttest to be administered as the last activity. 

3. The evaluator will complete the observation schedule on the first 
and third rnorning and the second and fourth afternoon. 

4. The questionnaire win be given the last afternoon before the 
posttest. 


7. Analysis 

\ 

• \ 

I 


1. A pretest analysis will be done indicating the participants' general 
level of knowledge about reading instruction prior to the workshop. 

2. A t test will be used to determine if a significant growth in 
participants* knowledge occup-red. C 

■3. A tabulation of observations will be made to determine the level of 
Involvement of workshop participants and the decree of congruency 
between the workshop plan and Its - implementation* 

4. The questionnaire will be tabulated to determine :he particioants' 
view of how well they understand the purposes of the new curricu- 
lum and how to effectively Implement it. 

5. Participants' attendance will be suinrarized to indicate the 
average daily attendance in relation to the number of teachers . 
scheduled .to attend the workshop. 

6. Workshop ^taff interviews will be summarized in narrative fashion. 


8. Report 

r 


1. An oral report by . the morning of the second day iridicatino the 
general ^reas of strengths and weaknesses of the participants 
based on their pretest results. ^ . 

2. An oral debriefing by the evaluator based on observation notes 
on the day after the workshop. , 

3. A written report to the project director, superintendent and 
School Board reporting the complete evaluation findinos regardino 
the workshop and the participants' suggestions for future trainino. 



ERIC 



Figure 4.14 

5 



Having completed each of the eight steps, you are now ready to assess 
the completeness of an Evaluation Plan by chp.i;king each of the eight questions 
originally asked about any plan. These questions are presented in the form 
of a checklist below to help you make sure an evaluation plan answers these 
questions for each objective or area to be evaluated: Use of this checklist 
can help you to insure that you have not overlooked important elements 
involved in the evaluation of your objectives. 



EVALUATION PLAN CHECKLIST 

Objectives Listed in Numerical Sequence 
From Your Proposal 







1 


2 


3 


4 


5 


6 


7 


8 


9 




1. 


What questions are being asked? 






















2. 


What information is needed to answer 
the question? 






















3. 


Where can the necessary information 
b"e~S3cured? 






















4. 


What can be used to find the answer? 






















5. 


Who will complete the instruments and 
what comparisons may be made? 






















6. 


When will ths information be collected, 
analyzed and reported? 






















7. 


What do we do with the data? 






















8. 


Who needs to know about it? 























Figure 4,15 



Timelog for an Evaluation 



The timelog, Figure 4.16, may be helpful in monitoring evaluation 
activities. The evaluation is divided into its major components. Under 
each component are various activities associated with that component. Next 
to each item is space for the date when that activity .should begin and when 
it should be completed. This can also serve as a checklist to make certain 
no oversight has occurred. 



SAMPLE EVALUATION TIMELOG 

■ 

.n-rrur^r. " COMPLETION 

ACTIVITY OATE DATE 



1. Purpose of the evaluation 

a. Clarify what and why the evaluation is for (i.e., pro- 
gram improvement, funding decisions, etc.) 

b. Clarify goals, objectives, questions or decisions 

2. Plan the evaluation 

Prioritize objectives or questions 
Determine information requirements 

c. Determine . information sources 

d. Select or develop appropriate instruments 

e. Select appropriate designs 

f;^ Schedule implementation z 

g. Select appropriate analysis techniques 

h. Determine, appropriate audiences for various aspects 
of information 

3* Implement the evaluation 

a. Assemble basel ine. data (.if needed) 

» 

b. Administer evaluation instruments as scheduled with 
the eva luation plan 

c. Collect process data on an ongoing basis 

d. Analyze evaluation data 

4, Reporting the information 

a. Report orally to proper individuals (if appropriate) 
b» Interim/progress reports 
c. rinal report 



Fi gure 4.16 



SWLE TIHELIHE 



TASKS 



1. Oetermine. what to, evaluate 



,2, Develop evaluation plans 
and instruments 



,3. Secure approval of 
instruments 



.4. Collect pretest data 

5. Analyze pretest data 

6. ■ Collect process data 

7. Collect posttest data 

Analyze process and 
product data 

9. Prepare final report 

10. Prepare' report sumry 
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Assessing the Quality of the Evaluation Plan 

So far we have discussed the elements of a comprehensive evaluation 
pla'n and provided a checklist. for monitoring the completeness of such a plan. 
A pI'smT^arT be complete and yet lacking in quality. The Checklist for Judging 
the Adequacy of an Evaluation Design developed by Sanders and Nafziger and 
displayed in Figure 4.18 is useful to apply in assessing the quality of an \ . 

evaluation plan. 



Sanders, J. R. and Nafziger, ,D. H. A Basis for Determining the Adequacy of 
Evaluation Designs , Portland, Oregon; Northwest Regional Educational . 
Laboratory, October 1975. pp. 9-T3. 
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CIIECKLI3T FOR JUDGING TlIE ADBQUACV OF AN EVALUATION DESIGN - ' 

DIriiciloflii For each quealloo below, olrolo whellwr the evalusllon desljn' hii clearly mel Ibe crIUrlon (Yee), bu clearly not met 4ho criterion (No), 
or cannot be clearly dolermlned (?) Circle NA If the criterion does not apply to the evaluation dealgn being reviewed, Use iiie Elalamtlon coluniiT 
(0 provide (urther explanation for criterion whore a No or a ? bns been circled. The word "program" will be used to moan tlie program, project or 
product being evaluated, , 



Title of 'Evaluation Document! 
Name of Devlowcri 



Criterion 




Criterion Met 


Elaboration 


1. Ci:llerla r/,:satdlng the Adequacy of the Evaluation 










' Conceptualization 










A. Conceptual' Clarity and Adequacy 










1. Is an adequate desorlptloa of the whole 










program presented? 


Yes 


No 


? NA 




2, Is a clear description given of the part of the 










program being evaluated? 


Yes 


No 


7 NA ' , . 




• 3. Is a clear description of the eyaluatlou approach 










given? (e,g,, comparison group study, slrjls • 






1 




group study, goal-free evaluation, foii^fsiatlve, 










summatlve, etc) 


Yes , 


No 


? NA 




4. Is the evaluation approach adequate and 










appropriate for evaluating the program? 


Yes 


No 


?' NA 





Daficd on tho above, do you fdel Uio evaluation Is cloarly 

and ttdctfualoly conceived? Yea No NA 



II. Scojie 

. I; Are Uie Inlonded outcomee or goals of ihe 



program clearly specilled? 


Yes, 


No. 


? 


NA 


'2, Is the scope of the evaluation brodd enough to 










gather Information concerning all specified program 










■ outcomes? .' 


Yes 


No 


? 


NA 


3. Are any likely unintended effects from the 










' ' . program described? 


Yes 


' No 


? 


NA 


<l. Is the approach of tho evaluation broad enough 










to include measuring these unintended effects? . 


Yes 


No 


? 


NA 


S. Is adc(|uolo cost Informatloa about the program 










included in the scope of the evaluation? 


Yes 


No 


? 


NA 


Based on the above, do you loei tlu) evaluation is 
adequate in scope? 


Yes 


No 


? 


,tfA 



Criterion Crilerlon Met 



C# Helevnnce • . 

I . 1. Are lh« audleocea fo^ Yes No ? 

I V . 2. Are the objoctlves of the evaluation explained? Yea No ? NA 
3. Are the objeollvefl of the evaluation congruent * 

wlth<;the Inlbrnui^loQ ae^^ \ ' 

' ttudlencea? Yea No ? NA 

/ ^ 4* Doea Iho information to be provided allow 
neceaaary declalona about the program or 

product to bo made? r Yea No ?. NA 



Oaaed on the above^ do you feel the Information provldod la 

relevant to and adequately servea (be neode of the Inten Yea No 7 NA 
audience? » ■ - 

D/ Floxiblll^ ; 
1« CflA tbe design be adapted easily to 

accCK)imoda(a chan^^ . Yes No 7 NA 

2. Are knd^^ 

evaluation dlflGusaed thoroughly? Yes No . 7 NA 

3, Cw useful Information be obtained In the 
face- of unforeseen conslralntSi' e.g. t non-* 

cooperolJon of control groups? Yob No 7 NA 

Sased on the obovoi do you fool thd evaluation study allows for 

lew Information needs td be met aa they arise? Yes No ? NA 



E, Feasibility 
I 1. Are the ovaluallon resburces (tlmop money and 
[ personnel) adequate to carry out tho projected 
\ acUvllies?" 

2i Are management plans specified for conducting 

the evaluation? 
3. Has adequate plaanlng been done to support tho 

'feasibility of conducting complex activities? 



lased on the abovci do you feel the evaluation can be 
tarried out aa planned? 



YeB 


No 


? . 


NA 


Yea 


No 


? 


NA 


Yes 


No 


.? 


NA 


Y08 


No 


? 


NA 
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Criterion 




Criterion Met 




Etaboradon- 


n. Criteria Concerning the Adequacy of the Collection 
and Proceesing of Information 


A, Hellabillly 










— 


1. Are data coUeotlon procedures descrltxsd well 












and was onre taken to assure minimal 












error? ^ ' - 


Yea 


No 


? 


NA 


- 


2. Are scoring or coding procedures 












objective? 


Yes 


No 


? 


NA 


, ■ 


a. A 1*6 the evaluation Instruments 












reliable? (he.; is reliability 


Yofl 


No 


? 


NA 




information Included) 












Dased on the above, do you feel that if the evaluatipn Vi. f; ^ 










* 


wore conducteil again the results would turn out the 












same? 


Yea 


No 


? 


NA 




D* Objectivity 












1, . Have attempts to control for bias In data 












collection and processing been 












described? 


" Yes 


No 


"? 


NA 




2. Are sources of Information clearly 












specified? 


Yofi 


No 


. ? 


NA 




3« Do theiblasoB of the evaluators preclude 












an objective evaluation? 


Yea 


No 


? 


NA 





Based on the above, do you feel adequate steps have been 
token to unsure objectivity In the various aspects of the 

evaluuUon? Yes No ? NA 



Figure 4J8 
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Critflrlon 



Criterion Met 



Elaboration 



C, nopreBentatlvenesfl 

" Are tho data collection InelrumontB valid? 
2. Are tho data colloctlon InBtrumentfl appropriate 

for the purposea of this evaluation? 
3« Does tho evaluation adequately address the 
queallona It waa Intended to answer? 


Yes 
Yes 

Yes 


No 
No 
No 


? 

. 7 
? 


NA 

NA 


Baaed on the abovei do you feel the Information 
collection and procesaing procedures ensure that the 
results accurately represent the program? 


Yes 


No 


? 




(J ■ 

0. Gonornllzabllily 

1. Are sampling leohnlquea adequate to permit 
genoralUatlona to the populullon of Interest? 

2. Oooa the cultural context of- data collecllon " 
techniques affect generalization? 

3. Are the Inferential statistics employed 
appropriate for the sample, data and tlio 
questions to bo answered? 


Yes 
Yes 

Yea 


No 
No 

No- 


? 
? 

? 


NA 
NA 

NA • 


Dascd on the above, do you fool the Information colieclod 
can bo generalized when necessary?-, ' 


yes 


No 


? 


NA . 


III. Criteria Concerning the Adequacy of the \ 
Presentation and Reporting of Information \ 


A. TImellnesB 

1. Have efficient reporting techniques been used 
to meet the needs of the clients? 
. 2. Does the time schedule for reporting meet 

1^ the needs of the audience? 


Yos 
Yea 


No 
No 


I 
7 


NA- 
NA 


Oasod on tho above, do you feel the information is timely enough 
to bo of use to the client? . ^^^^^..—^ 


Yea, 


No ' 


i 


NA . • 



Qrj Figure 4.18 



Criterion Criterion Met EJoborjOton 

■ : D« Pervaalvoneaa 

1. I8 Information dlfleemlnalod to all \ - 

Inteindod audiences? Yob No ? NA 

2. Aro contractuoS constraints on dlflsemlnation 

i:.'; of evaluation 1^^^^ Yes Ho ? NA 

3. Are attempts iielng mode to make the 
'evaluation In/ot^mallon available to relevant . 

audiences beyond those Bpocined In the , ' 

• contract? " . Yes No 7 NA \. 



Baaed on the above, do you feel that Information la 
being provided to all wlio noed It? 


Yea 


No 


. ?' 


NA 


' ' ■ . V: ■,; . 


[V, Qenerfll Criteria c> 










■ \' 


. 'A. EthicQl Coneldoradons ^ . , 
7 Do test admlnletratloo proceduroB follow 

profosalonal standards :of ethics? 
2, Have protection of human subjects 
guidelines been followed? 
' 3/ lias conlldeniiaUt}^ of data been guaranteed? 


Yes 

Yes 
Yes 


No 
No 

No ^ 


? 

?■.,: 
•? 


NA ■ 

' NA 




hsed on the abovai do you feel the evaluation study <v; 
trlctly follows professional standards of ethics ? 


Yes 


No 


? 


NA 




n. Protocol 

U Aro appropriate persons contacted in the 

appropriate sequence? 
2. Are Department policlet tr4 pfoccdures 

(o bo followed? 


Yes 
Yes 


No 
No 


?. 
? 


NA 
NA 


1 


lased OR tfic ohovo, do you feel appropriate protocol 
itepa were planned? 


Yes 


No 


? 


NA 
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Implementing the evaluation 



How Do You Collect and Analyze the Inf oriTiat io.i ?. 
Now that you have prepared an evaluation desitji you' 
are left with the task of implementing it. This 
section will provide you with Some hints to use in 
"collecting and processing information. 



• . -■■ ■ -' . - ' ■ , i . ■ • ■ 

Collecting Baseline Data ' 

Basel ihe dataw are data which describe or reflect the students' level • 
of development, achievement or attitude before they experienced the" program 
or process you are evaluating. This may be information from previous tests 
or previous year' s, work,, attendance, etc. . If information such as growth or 
change in specific students is needed, 'baseline data will provide their 
starting point. . .. 
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The collection of file data as a baseline is generally easy since the 
'information is already availably However, certain preliminary arrangements 
need to be made, such as: / - • - 
V • identifying where each set of- data is located 

• determining who, has authorized access, to the files 

• preparing a form for recording air of- the data to be collected 

' ■ t training and supervising the persons collecting" the data (if other 
than yourself) 

» scheduling time for the data collection _ 

.... ' ' ^ .\ 

Points to Keep in Mind When Testing 

• ... 

^- Here'are so(fi' important points to keep in mind when planning and admin- 
istering tests-. 

Explain Your Purposes ." Be sure to tell students why you are testing 
and encourage them to do th'eir^ best. Let €hem know if the test results will 
affect * them in some way such ,as for selection of program participants or for 
individual student counseling. ' 
' Environment and Other Considerations .. Test adrffinistration should be 

-uniform for all groups of students tested* Directions for administering 
each of the evaluation instruments should be studied by the testers prior ' 
to the day of testing since these may affect the size of the group to be 
tested at one time, room conditions or the-: need for special materials (such 

• as stop watch, i.f a tfmed test is being used). 

The testing ibceition should be well lighted and have adequate ventila- 
tion. There should be sufficient testing space for each .student, both for 
comforf and to minimize the possibilities of copying. The same test 
instructions should be used for all groups of- students tested and the timing 

,of testing sessions, where appropri,3te, should be strictly followed. Each - 
student should have a sharpened '<o -i- lead pencil when Digitek or other 
machine-readable answer sheets are used. It is a good idea to provide these 
sharpene;d pencils and collect them with the test booklefe^and answer sheets. 
Extra pencils should be available if needed. \ . * ,^ 

tCareful monitoring during the testing period is necessary, especially 

w,>ic»n special answer 'sheets are being used. It is important that students 

, understand how and where to mark answers since only portions -of some answer 



shf^ets are used and there are wide format differences among answer sheets. 
During this time the examiner can also check to see that the correct pencils 
are being used since marks made in ink are not readable by machine. 

When students complete the tests, examine them immediately for missing 
sections, mul ti pie' responses and lack of identification. In this way the. 
difficult problem of gcfing back to the students can be avoided. 

Follow-up . Immediately after student testing, it is useful for the 
test administrator to write down any irregularities that mc;,)^ nave occurred 
~dartng t h e te s L h iy-perrod-; — Th4sHfiformaEti<)r^-i^-^^ -1 -h(:-)'pi;;g - to-inter^---.- 
pret unusual results. Scoring and coding of evaluation instruments should 
begin as soon as possible after the instruments are administered. This 
minimizes loss of data and insures a quicker availability of test results. 

Processing Data 

You need to decide whether program data are to be scored and/or coded 
within the district or csipntracted to a person or agency outsfde of the 
d'istrict. If the dataware scored or coded within the district a decision 
is needt!5d . as"T5 whether it will be done manually or by macrvlr-^ 

A decision whether to contract with a person or agency outside the 
'district for scoring and coding services is influenced by factors such as: 

• The availability of trained personnel within the district to provide 
thesa services 

• The availar ility of necessary machine scoring facilities viithin the 
district - ^ 

• The ^availabil i ty of funds for an outside contract 

• The turnaround time of either service 
•The costs of either service 

^ The accuracy likely under either service 
• A decision as to whether instruments should be scored manually or by 
machine is influenced by factors such as: 

t The number of instruments to be scored 

• The availability of traUjc* personnel, equipment and programs for 
scoring the \instrument\ 

• The turnaroun)fi time of either system 
% The costs of either system 
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REPORTING EVALUATION FINi:|iNGS 



How Should I Report the Evaluation Findings ? First 
reciall who the primary audiences are for the report 
and the types of questions the audiences are likely 
to have about the findings. This will help deter- 
mine when to give the repor^t, the level of presen- 
t?\tion, and appropriate methods for reporting. 



Timing 



/ 



Remember: 

A report limited in scope but presented prior to making program 
decisions is preferable to a comprehensive report presented two 
weeks after important program decisions have been made. 
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Purposes 

■ ' ^ 

Since we saw in Section 1 that evaluation can serve multiple purposes, 
it is natural that the same data may be reported at different times and in 
different ways. - 

For example, the Woodcock Reading Test used in cur earlier examples can 
illustrate this point. If individual student pretest data were returned to 
the respective classroom teachers, they might use the results to place students 
in one of three homogeneous reading groups, to determine which students may 
need a -more in-depth diagnostic test and to determine what particular strengths 
and weaknesses their class as a whole has in reading in order to gear reading 
instruction more appropriately. To do these things, the-results would have 
to be made available to teachers as early as possible after the tests were 
administered. 

If the same test had been used in prior years and. a new reading series 
were introduced this year, the teachers would probably be interested in com- - 
paring student growth between the current and prior year results. For this 
purf^ose they would probably want to receive the evaluation report prior to 
the one Week teacher planning conference in August- in order to consider modi- 
fications in their reading program for the coming year. 

Inte: irn Reports 

Interim reports are particularly useful for providing feedback to the 
project staff that they can use in refining a program's operations. For 
example, classroom observations during the first two months of the school 
year may reyeal that students^are usually not completinig. assignments properly 
because the directions are inadequate. A tvm^y interim report addressing 
this issue could allow, the project staff to rewoird or expand other written 
directions so that students could more profitably complete future assignments 
during the reniainder of the\school year. 

Interim reports are also useful if the decision to continue the funding 
of a project'must be mede prior to the availability of a completed end-of- 
year evaluation report. If you know that such a funding decision were to be 
made in late.Apri'i, you might develop an evaluation design in which all pro- 
gram students were pretested in September, a random half were^ posttested in- 



in iate March and the other half in mid May. The design might also call for 
clas-.-rooni observation between- January and March to focus on the quality of 
proar^sm .inplernentation and on teacher questionnaires or interviews .in March 
to record the teachers* perceptions of the program and their recommendations 
regarding Its improvement. 

How Much Detail to Include 

Trie question of how detailed or specific to make a report also relates 
back- to an understanding of the audience. If the report is to be given 
about individual classrooms, teachers are usually interested in specifics.. 
If the report contains data for an entire district, it may be best to simply 
summarize it and place the details in an appendix. The same data may be 
reported to various audiences at various levels of specificity. For example, 
a complete evaluation report may be a summary description of the instruments 
used to measure a particular objective, the student population or sample,, 
procedures used to analyze the'data, the statistical findings and a discussion 
of the findings related to that objective. Project staff may be primarily 
in. "^s ted ''in a discussion of how recommendations could be used to improve 
the program. Parents or community persons may simply be interested in a 
single sentence summarizing how students performed on a particular objective. 

A written evaluation report should be organized so that various 
audiences can quickly assess the information of particular interest to them. 
If a program evaluation report is intended for the funding agency, don't 
forget to see what "^orms or report format they might require or suggest. 
A moderately detailed table of contents can be very helpful to the reader. 

Report Format 

Effective evaluation reports can occur in both oral and written format. 
Oral reports are particularly appropriate as initial feedback from visiting 
consultants or site review team members. Such a debriefing can allow for 
a sharing of the observer's first-hand perceptions and can facilitate a 
useful exchange between the observer and project staff. Thus, when an oral 
debriefing occurs it is helpful to have the observer react to some questions 
or issues that served as the focus for the site visit as well as to share 



his or her perceptions regarding unanticipated outcornes of the program.- 
Reactions by the project staff to the observer's connments are also important 
before discussing alternative recoimiendations that may arise. 

A comprehensive written evaluation report usually contains the 
following sections: " ; 

• Executive Summary (a 3- to 10-page overview of the report findings 
and recornnendations) 

• Introduction (identifies the purposes and audiences for the report, 
provides dn _overview of the c ontents and describes any disclaimers 

if need^rd) ^ — — _ _ _ 

• Program Description (can contain some of the information in 
Chapter 2 of this manual) 

• Objectives/Questions to be addressed by the evaluation 

Description of Evaluation procedures, designs and instruments 
(in summary fashion here with more details, if needed, in the 
appendix) • 

• Discussion of Findings (It is sometimes useful to organize the 
findings, first around the instruments used and/ secondly , around 
the evaluation questions or objectives.) 

• Conclusions and/or Recormiendations (Enough information should be 
given. here to allow the reader to see-the rationale and data 
base for any recommendations.) 

• Appendices (Separate appendices containing locally developed 
instruments, technical data and detailed tabulations , can make the 
body of the report less technical.) 



Report Summaries 



Since many evaluation consumers do not have the time or interest to 
read a thick or technical report, it is important to communicate a summary 
of the findings. An ex^utive summary is^an important element in a total 
evaluation report. A separate evaluation digest (of perhaps five to fifteen 
pages) and an evaluation abstract (of one or two pages) are also handy and 
useful for parents, legislators, board -members and others interested in an 
overview of the findings. 

. The executive summary, if p^resented as an introductory chapter to a 
comprehensive report, can include both the findings and suggestions concern- 
ing what sections of the total report may be of special interest to certain 
audiences* The evaluation digest and abstract, on the other hand, should 



each be written as self-contained documents that do not refer to specific 
sections of the total report. While' the evaluation digest may/ cite the 
findings and reference them to the types .of evaluation instruments and pro- 
cedures used to produce them, the abstract usually includes only the major 
findings and recoimendations without attempting to document them. 

Utilizing Evaluation Findings 

Before deci d ing w hat actio n s— to tak e on t h e~-ba^4s-o^-a^-Comp4>€-t^ 

evaluation report some administrators have founr" 't a useful practice to 
obtain a s.eparate technical critique of the report from another evaluator,. 
This outside review gives them a better j'dea of how reliable the evaluation 
procedures were. This in turn can help them determine how much to trust 
the- findings . Other administrators request a separate informal report of 
the. project staff , thus obtaining their perceptions of the program. 
Reactions of the project staff to the evaluation report itself are also 
useful before taking action on any of its recummendations. 

Sometimes the persons conducting an evaluation may do an excellent job 
of collecting arid reporting program findings and yet not be expert in 
preparing recommendations. Therefore some administrators will call in 
knowledgeable people to review the evaluation findings and then propose a 
set' of recommendations based on their content knowledge of the program. 
Sometimes a conference is then held of project staff, evaluators, community 
members, outside consultants and other district staff t-: jointly review and 
discuss recommendations* Such a procedure gives the broadest base for 
determining important changes that may be made in a project. 

Closing Statement 

We hope that this book has helped you to understand better how to plan, 
carry out and use evaluation. Although your evaluation efforts may not be 
as technical or complex as a university research study, they should be 
f^andled in such a way as to -have meaning and utility in improving your pro- 
g\ms or activities. Your understanding of evaluation will increase as you 
app^X^to problems facing your school or district what you have learned. 
Good luckl 
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Anderson, S,B., Ball, S., Murphy, R,T. & Associates. Enci/clopedia of ^ 

Educational Evaluation . San Franci SCO, Cal ifomia : Jossey-Bass, Inc., 
1975. - 

This encyclopedia contains over 100 short articles covering many 
aspects of evaluation including evaluation models, functions of 
evaluation, program objectives and standards, social context of 
evaluation, design, mea'surement approaches, and analysis and 
interpretation. The articles are written at an introductory 
level and provide references for'more ^'n-depth treatment. Admin- 
istrators, will find this encyclopedia eful way of obtaining 
introduction to evaluation topics o'^" spec '^1 interest to them. 



Campbell, D.T^, S Stanley, J.C. Experimental and Quasi-Experimental Designs 
T for Research. Chicago, Illinois: Rand McNally College Publishing 
Company, 1963. I -^1 

Long considereid the bible of research and evaluation design, this 
paperback examines systematically the validity of 16 experimental de 
. designs against 12 common threats to valid inference. The cautions 
identified should help you in selecting an appropriate design for 
your program evaluation. 
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Gottman, J.M., & Clasen, R.E. Evaluation in Education: A Practitioner's 
" Gtiide. Itasca, Illinois: F.E. Peacock Publishers, Inc., 1972. 

The authors have developed a programmed textbook to assist the 
reader in learning how to: do a needs assessment, write measur- 
able objectives and design measurement procedures, flowchart, a 
and design and use quality control procedures. The use of 
time-series analysis is given much attention. The appendix 
"^explains a number of basic statistics.. 



Isaac, S., & Michael, W.B. Handbook in Research and Evaluation. San Diego 
CalrF ornia: Robert R, Knapp^ iq71 . 



This is a well organized, easily accessed reference. It is quite 
brief but. touches on most topics related to program evaluation, 
such as planning evaluation studies, research designs, measure- 
ment, statistical techniques and data analysis. 



Lyman,' H.B. rest scores and what They Mean. Englewood Cliffs, New Jersey: 
Prentice-Hall, Inc., 1963. 

This readable paperback describes some basic attributes of a 
test (validity, rel iabil ity and usability), some elementary 
test statistics and ways in which test results can be used 
appropriately. HelpfuT ideas are also presented on ways to 

Mager, R.F. Preparing instructional objectives ; Belmont, Cali-fornla: 
Fearon Publishers, Inc., 1975. _ * ^ 

It has become a classic in a very short time as a "how to" book 
on, preparing instructional objectives. It is understandable 
and contains many good examples.- 



Stufflebeam, D.L & et al. Educational Evaluation S Decision .Making. 
Itasca, Illinois: F.E. Peacock Publishers, Inc., 1971. 

■ This book was written by a seven-member Phi Delta Kappa National 
Study Comnittee on Evaluation. It contains an in-depth analysis 
of educational evaluation springing from the authors' comprehen- 
sive definition of evaluation. Administrators may be particu- 
larly interested in the chapters dealing with educational decision 
making and with the one on organization and administration of 
evaluation units in school districts. 



Vargas, J.S. writing worthwhile Behavioral Objectives . New York, New York: 

Harper & Row, Publishers, Inc., 1972.. 

This is a good follow-up to Mager' s book described above. Once . 
an objective is prepared, it should be judged as to its worth- 
^- ' whileness. This is an easy-to-read book which can help you 
assess the worth of specific objectives. 
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Webb, E. , Campbel l r D. Schwarts, R. , & Sechrest, L. unobtrusive Measures: 
Non^reactive Research in the Social Sciences . ChlGago, Illinois: 
Rand McNallyS Company, 1966. • - 

The title is more ominous than the contents. Often when we . 
^ . measure things in the classroom, the teacher or students change 
as a result. Unobtrusive measurements are designed to minimize 
this effect i. This book describes a number of unobtrusive pro- 
cedures that can be used in program evaluation. 

\ Worthen, B.R., & Sanders, J. R. Educational Evaluation: Theory and Practice. 
^-.^Worth j n qton.,^ Oh i o : C ha ries A . Jones Publishing Company, 1973.* 



This boolc^ of readings is particularly helpful in providing the 
reader with frameworks for planning evaluation studies. Various 
evaluation models are described fay their authors and each is 
followed by a paper describing how the model was actually applied. 
Administrators may be particularly interested in the chapter deal- 
ing with the relationship between the evaluator and the decision 
maker. ■ 
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Objectived and Goals , 

■■ Mager, R.F. Developing Attitude Toward Learning. Belmont, California- •. 
Fearon Publishers/Lear Siegler, Inc.; '1968. 

Mager, R.F. Preparing instructional, objectives . Belmont, California* 
Fearon Publishers, Inc., 1975." 

■Popham, W.J., Eisner, E.W., Sullivan, 'h. J. , & Tyler, L.L, "Instructional / 
Objectives." AERA Monograph Series on Curriculum Evaluation. No. 3./ 
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